Towards Trusted LLM based Curator Agents

Motivation: TAMMS KG

Activity Specific Agents: Visual Agents

• “Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.
• Hu, Ziniu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A. Ross, Cordelia Schmid, and Alireza Fathi. 2023. “AVIS: Autonomous Visual Information Seeking with Large Language Model Agent.” arXiv.

Visual Agents Architecture: Different LLMs based on Role

• “Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.
• Hu, Ziniu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A. Ross, Cordelia Schmid, and Alireza Fathi. 2023. “AVIS: Autonomous Visual Information Seeking with Large Language Model Agent.” arXiv.

Activity Specific Agents: Visual Agents Transition Graph

• “Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.
• Hu, Ziniu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A. Ross, Cordelia Schmid, and Alireza Fathi. 2023. “AVIS: Autonomous Visual Information Seeking with Large Language Model Agent.” arXiv.

We need to think through what Trusted Means!

Starting with a CSV (Navy Maintenance Data)

Korini, Keti, and Christian Bizer. 2023. “Column Type Annotation Using ChatGPT.” arXiv. http://arxiv.org/abs/2306.00745.

Context Matters!

Converting Legacy Enterprise Data into Knowledge Graphs with AI and JSON LD | Eliud Polanco

JSON-LD as a Bridge

Converting Legacy Enterprise Data into Knowledge Graphs with AI and JSON LD | Eliud Polanco

How can we create “Semantic Microservices”

ChatGPT “Plugin” Architecture as Example

• ChatGPT plugins (openai.com)
• Introduction - OpenAI API
• About Get a ChatGPT plugin up and running in under 5 minutes!
• Use LangChain OpenAPI
• ChatGPT Plugin – LangChain
• How to build a tool-using agent with LangChain

Example Service – Retrieval Augmented Generation (We’re not doing this yet!)

• Experiments in extracting tables from navy 3-M manual for OPNAV 4790/2K data structure Resources
• Sample KG construction using OPNAV forms 4790/ 2K as a schema template
• Repository for formatting the Joint Electronics Type Designation System for ML and KG Usage